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EXECUTIVE SUMMARY 



The U.S. Department of Agriculture (USD A) sponsored the National School Lunch Program 
Application/Verification Pilot Projects to test ways to improve the process for certifying students 
for free or reduced-price meals. This report presents findings of an analysis of pilot project 
operations and costs for two alternatives to the current application-based certification process — 
Up-Front Documentation and Graduated Verification — that were tested in 12 public school 
districts over a three-year period from 2000-2001 thru 2002-2003. 



BACKGROUND 

Millions of U.S. children participate in the National School Lunch Program (NSLP) each 
day, receiving free or reduced-price lunches that make an important contribution to their overall 
nutrition. Concern has mounted, however, that many of the children certified as eligible may in 
fact be ineligible because their family income is too high. Under the existing eligibility process, 
families must state their income on the application for the program but do not need to submit 
additional documentation. Districts select a small sample of applications for income verification, 
which is done later in the year. 

To address whether the eligibility process could be made more accurate, USDA sponsored 
pilot projects testing two new approaches to certifying eligibility: (1) Up-Front Documentation 
(UFD), and (2) Graduated Verification (GV). 

Districts using UFD required families to document their monthly income or receipt of public 
assistance when they submitted their application for free or reduced-price lunches. Districts then 
used this documentation to make an eligibility determination, but they did not verify any 
approved applications later in the school year. 

Districts using GV allowed families to use the standard application process, which does not 
require income documentation, but changed key aspects of the usual verification process. After 
verifying a small sample of approved applications, these districts conducted additional 
verification if 25 percent or more of the applications in the initial test resulted in benefit 
reduction or termination. Depending on the findings of the second round, a third round of 
verification was sometimes required. Another feature of the GV approach was that, if a family 
lost eligibility due to verification activities in any given year, in the subsequent year the family 
had to supply verification materials at the time of application (usually the start of the school 
year). 



STUDY DESIGN AND METHODOLOGY 

The main study used a comparison design to select additional districts that were not 
participating in the three-year pilots but had similar economic characteristics and geographic 
locations. The evaluation of UFD included nine pilot districts and nine matched comparison 
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districts. The evaluation of GV included three pilot districts and three matched comparison 
districts. 



The analysis of operations and costs uses data from interviews with school district 
administrative staff who were involved in application and verification activities at the pilot and 
comparison districts. We used interviews with staff to learn how key activities were organized 
and carried out and how operations changed because of the demonstration procedures. District 
staff also provided their estimates of the time required to perform application and verification 
activities, which form the basis for cost estimates. 



FINDINGS ON OPERATIONS AND COSTS 
Up-Front Documentation 

The UFD pilot districts implemented the pilot as planned. Staff implementing the new UFD 
procedures believed that the pilot improved the accuracy of income reporting, largely by 
relieving fa mi lies of the need to distinguish between gross income and net income — families 
could simply provide pay stubs from which school staff could determine gross wages. In 
addition, district staff felt the process was fairer, because all families, not just a small sample, 
were subject to the documentation requirement. 

A major challenge in operating this form of the pilot was that pilot districts received more 
initially incomplete applications (because documentation was required with the application) than 
comparison SFAs operating under standard eligibility determination procedures. Staff also noted 
that the documentation requirement did not prevent families who wanted to conceal some of their 
income from reporting and documenting some income sources while failing to report others. 

Pilot project staff reported that UFD created some additional work in (1) following up on 
incomplete applications, and (2) making eligibility determinations based on direct income 
documentation. In general, however, they reported incorporating these additional activities into 
their application processing with only modest additional burden. Our formal cost estimates 
imply that UFD created a modest increase in application-processing costs per applicant. This 
increase in cost per applicant was fully offset by the reduction in the number of students for 
whom applications were received and approved. Thus, overall costs for eligibility determination 
were unchanged by the UFD pilot. This cost neutrality came at the expense of targeting 
efficiency, however, because the reduction in certification was all due to reduced certification 
among eligible children. 



Graduated Verification 

Compared to the UFD model, the GV model was more complex to operate and more 
burdensome for staff to implement. In addition, the logic of Graduated Verification’s objectives 
was not as clear to the staff responsible for implementing the procedures. GV was complex, 
because it required one, two, or three rounds of verification, depending on whether the 
percentage of cases reduced or terminated at each stage was above 25 percent. In most districts. 
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either two or three rounds of verification were needed, thus increasing costs compared to regular 
NSLP procedures. Another result of this design was that the workloads, especially the large 
workloads associated with the second and third rounds of verification, were not easy for School 
Food Authority staff to predict or plan for. Staff had to find time in already tight schedules to 
carry out these later rounds, if they proved necessary. 

GV was also more complex than standard NSLP procedures at the application stage. This 
occurred because most families were subject to the standard verification procedures, but some 
were subject to the same process as all families followed in the UFD pilots. Furthermore, the 
number of households required to provide documentation varied from year to year and depended 
on the number of rounds of verification conducted in the prior school year. Careful record 
keeping was necessary to support these determinations. 

Because of these complexities, GV was generally not implemented with the same degree of 
fidelity to the original pilot model as was the simpler UFD model. Indeed, three of the four 
original GV pilot districts did not implement the model completely for the three school years of 
the demonstration. 

Reflecting the complexities of the GV model, our analysis of costs indicated that both the 
cost per applicant and the total cost per enrolled student increased as a result of GV. These 
estimated changes in costs are substantial — in excess of 50 percent of the base cost of processing 
applications and conducting one round of verification. 
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I. INTRODUCTION 



The National School Lunch Program (NSLP) and School Breakfast Program (SBP) serve 
nearly 4 billion free and reduced-price meals annually to children certified as being from low- 
income households (U.S. Department of Agriculture 2003). In recent years, however, 
policymakers and the public have raised concerns about the integrity of the programs’ process 
for establishing eligibility for these benefits. In response, the U.S. Department of Agriculture 
(USDA) asked school districts around the country to voluntarily participate in the National 
School Lunch Program Application/Verification Pilot Projects to test ways of improving the 
process for certifying students for free and reduced-price meals. USDA published a report on the 
experience of pilot districts in the first year of implementation. It contracted with Mathematica 
Policy Research, Inc. to conduct an evaluation of two of the approaches that were tested: (1) Up- 
Front Documentation (UFD), and (2) Graduated Verification (GV). 

This report presents the results of an analysis of the operational aspects of the pilot projects, 
including the procedures used to implement the pilot policies and the costs associated with these 
procedures. Three companion reports describe other key findings of the evaluation: 

• Impacts of UFD and GV on the certification of eligible and ineligible students for 
free or reduced-price benefits (Burghardt et al. 2004) 

• Impacts of UFD and GV on whether certified (and noncertified) students actually 
received school lunches (Gleason et al. 2004) 

• Impacts of UFD and GV on rates of application of eligible and ineligible students for 
free or reduced-price benefits and analysis of household income reporting and SFA 
application processing (Hulsey et al. 2004). 

The rest of this chapter presents background information for the study and discusses the key 
research questions addressed. 
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A. STUDY BACKGROUND 



Several studies examining income levels of students certified for free or reduced-price meals 
have found that a nontrivial number of these students have income levels that make them 
ineligible for the level of benefits they are receiving (e.g., U.S. Department of Agriculture 1990; 
and U.S. Department of Agriculture, Office of Inspector General, 1997). To address this issue, 
several school districts began testing alternative ways of determining the income eligibility of 
students’ families, and this evaluation focuses on a subset of these districts. In particular, this 
evaluation includes nine districts that tested UFD during the 2000-2001 through 2002-2003 
school years and three districts that tested GV during these same years. 

Under UFD, districts required that all applicants for free or reduced-price meals provide 
documentation of their income or food Stamp/Temporary Assistance for Needy Families (TANF) 
receipt with their application.^ If the application did not include documentation, a student could 
not receive benefits. After the district reviewed and approved applications, it was not required to 
perform the verification of income for the small sample of households called for in federal 
regulations. Students approved through direct certification in the UFD pilot districts were not 
subject to these requirements, which applied only to students who submitted an application. 

Under GV, application procedures were strengthened and, in certain circumstances, the 
verification process was enhanced. First, households who applied for free or reduced-price 
meals and whose benefits had been terminated or reduced in the prior year because of the 
district’s verification procedures had to provide documentation of their incomes or of their 
categorical eligibility at the point of application. Second, the district had to conduct the standard 

*For additional details on the pilot projects and how their rules differed from standard 
district eligibility determination procedures, see Burghardt et al. (2004). 
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verification of three percent of participating households and the following additional 
verifications: 

• If 25 percent or more of the originally verified applications led to a termination or 
reduction of free or reduced-price meal benefits, the district was required to verify an 
additional 50 percent of remaining applications. 

• If 25 percent or more of these second-round verifications resulted in terminations or 
reductions in benefits, the district was required to verify all remaining applications. 

B. RESEARCH QUESTIONS 

This report focuses on two aspects of the pilot projects: 

1. We describe the procedures the pilot districts followed to carry out the new policies 
put in place as a result of the demonstration and report on perceptions of SFA staff 
about the procedures. 

2. We estimate the administrative costs of carrying out these procedures. 

Examining the first of these research issues provides important background for interpreting 
the estimates of net impacts presented in other study reports (Burghardt et al. 2004; and Gleason 
et al. 2004). In particular, while one objective of the pilot projects was to deter certification 
among ineligible households, we found no measurable impacts of either set of pilot procedures 
on the certification rates of ineligible households. This raises the following question: Were there 
no impacts because (1) the pilots procedures were not carried out as planned, or (2) the 
procedures were carried out as planned but were not effective in achieving lower certification 
rates among ineligible households? Put another way, the implementation analysis allows 
policymakers to reach an informed judgment about whether the demonstration was a meaningful 
test of the interventions that USDA envisioned when it mounted the demonstrations. 

In addition, examining the first issue highlighted above offers insights from the experience 
of the staff at pilot districts that can help policymakers as they consider whether and how to 
adapt the procedures tested in the demonstration for further testing or broader implementation. 
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The experiences and perceptions of the district-level staff who implemented the pilot procedures 
can also provide valuable insights for improving the policies. 

Examining the costs of the pilot procedures is important in supporting an overall assessment 
of the pilot policies. It provides information with which to weigh the costs and benefits (both 
monetary and non-monetary) of the policies that were tested. 
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II. METHODS USED IN EXAMINING THE PILOT PROJECT PROCESS AND COSTS 



In this chapter, we deseribe the researeh approaeh used in the analysis of proeess and eosts. 
We diseuss both data eolleetion and analysis methods. 

A. RESEARCH DESIGN OF THE OVERALL EVALUATION 

We begin our diseussion with an overview of the overall design of the evaluation, sinee our 
approaehes to examining program proeesses and eosts for the pilot distriets substantially refleet 
this design. Here, we deseribe the general design, then note several ways in which the methods 
used in this report differ from those used for other parts of the evaluation. 

USDA seleeted the pilot districts participating in the study from applieations submitted in 
response to a Federal Register notiee inviting distriets to submit proposals. The eharaeteristies 
of these pilot distriets are deseribed in Burghardt et al. 2004. Both the UFD and GV pilot 
distriets ineluded in the evaluation were eoneentrated in the Midwest and Northeast. They also 
were more likely to be in suburban loeations and less likely to be in rural areas than the average 
distriet in the eountry. The largest pilot distriet enrolled just under 8,000 students. In general, 
the UFD districts had lower levels of child poverty and higher pereentages of students who were 
white, non-Hispanie than the typieal distriet nationwide. In eontrast, the GV distriets had higher 
levels of ehild poverty and lower percentages of students who were white, non-Hispanic than the 
typieal distriet nationwide. While the partieipating distriets represent a relatively broad eross- 
seetion of the United States, no sehool distriet with more than 10,000 students submitted an 
applieation to partieipate. 

For the evaluation, we paired each of the participating school districts with a comparison 
district. Each of the eomparison distriets was ehosen to be as similar as possible to its matehed 
pilot, exeept that it had not implemented the demonstration policies. Key faetors in ehoosing 
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these comparison districts included location relative to the pilot project, size, ethnicity, and 
NSLP certification procedures. 

Burghardt et al. (2004) includes a more detailed discussion of the matching process and the 
characteristics of comparison districts. Overall, Burghardt et al. (2004) concluded that the 
matching process was reasonably successful in identifying a group of districts that were similar 
to the pilot districts. However, given the small number of districts involved — nine UFD and 
three GV districts — there is no assurance that matches will be very close on any individual 
variable. The small-sample-size problem is especially acute in the process and cost analyses, 
where the unit of observation is typically the whole district and one or a few staff members (as 
opposed to the unit of observation in the impact analysis, which is the student or the student’s 
household and where the sample size exceeded 3,000 observations). The impact analysis for the 
evaluation (Burghardt et al. 2004) exploited the comparison design by comparing differences in 
key outcomes — for example, the percentage of children from families with incomes above 185 
percent of poverty who were certified for free or reduced-price meals or the percentage of 
children in this group who obtained a free or reduced-price NSLP lunch on a typical school 
day — among representative samples of students in the pilot and comparison districts.^ Similarly, 



*For some measures, we also examined changes in key outcomes (such as the percentage of 
students actually receiving NSLP lunches on a given day), using administrative data that school 
districts routinely compile as part of their program operations. This design is often referred to as 
a “double difference” design because it compares differences between pilot and comparison 
schools in changes between the demonstration period and the baseline period before the pilots 
started. This is often viewed as the strongest available design, short of random assignment, 
because the focus on changes implicitly controls for various characteristics that remain 
reasonably constant over time. Using a full double difference design was not possible in the 
implementation and cost analysis, however, because the necessary baseline data were not 
available. In particular, whereas districts routinely keep the data needed for the impact analysis 
as part of their compliance with NSLP reporting regulations, the districts are not required to 
maintain information on procedures and costs at a fine enough level of detail to support the 
process and cost analysis work. Furthermore, because the contract for the current evaluation was 
awarded only after the pilots had begun operations, it was not possible to obtain pre-pilot 
baseline data as part of the research. As discussed later, a limited amount of retrospective data 
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in this report, we rely primarily on contemporaneous comparisons between pilots and their 
comparisons, as well as on detailed analysis of specific work activities that we know the pilot 
procedures affected (or sometimes created). Next, we describe these approaches in detail as they 
apply to the two parts of the process and cost analysis. 



B. PROCESS STUDY METHODS 

To collect data on the processes used in the pilot projects, we conducted telephone 
interviews with staff responsible for NSLP application and verification work in the 25 study 
districts. We developed semistructured interview protocols that covered the following topics: 
(1) the steps of the application and verification process the districts used; (2) the type and level 
of staff involved in each step; (3) the time frame in which each step occurred; and (4) how 
procedures differed from those used in 1999-2000, the year before the pilot began. In pilot 
districts, we also asked about respondents’ perceptions of the pilot, including challenges 
encountered, solutions tried, and the perceived overall effects of the pilot. 

We examined in detail the experience of pilot sites in implementing the demonstration 
procedures to assess whether UFD and GV were carried out in the way USDA intended. To 
assess how the pilots affected the districts’ processes and perceived workload, when possible, we 
explored how pilot procedures differed from (1) the procedures the same districts used before the 



(continued) 

were obtained, but they were not sufficient to support a full double difference evaluation 
approach, both because of lack of detailed records and because of turnover in the school 
personnel who could supply information. 

2 

The study included 25 districts because 1 1 pilot districts each had one matched comparison 
district, while one GV pilot district had two comparison districts. For the pilot-comparison 
district pair in which two comparison districts were used, the estimates for each comparison 
district were given a weight of .5 in calculating the comparison district total for this pilot- 
comparison district pair. 

-5 

Two districts began pilot operations in school year 2001-2002. The pre -pilot period for 
these two districts was school year 2000-2001. 
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pilots began, and (2) the procedures currently used in comparison districts. In some districts, 
however, it was not possible to ask questions about the procedures in 1999-2000 because the 
staff member(s) available to be interviewed had not performed NSLP work at that time."^ We 
used three steps to determine which staff members were primarily responsible for conducting 
NSLP application, approval, and verification work in each district and would therefore be most 
appropriate for the interviews. First, we sent an e-mail to the business administrator or 
superintendent at each district, explaining the process and cost analysis and indicating that we 
would like to conduct in-depth interviews with district- and/or school-level staff responsible for 
NSLP certification and verification work. Next, we followed up the e-mail with a telephone call 
to the same school official and asked for the names and contact information of the appropriate 
staff who conducted certification and verification work. Finally, we called the staff members 
who had been identified, explained the process and cost analysis, and set up appointments for 
telephone interviews. 

In most districts, we interviewed a single staff member, although it was sometimes 
necessary to interview other staff members. We spoke with more than one staff member in 
districts where two people shared primary responsibility for NSLP application processing and 
verification, either both having responsibility at the district level, or one at the district and the 
other at the school level. In addition, we interviewed more than one staff member in districts 
where the person responsible for conducting application and verification work had changed just 
before or during the pilot, and the former staff person was still available to be interviewed. 



'^In these cases, we tried to contact the person who had been in charge of applications and 
verification work in 1999-2000, but this was not always possible. 
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C. COST STUDY METHODS 



Most districts do not keep sufficiently detailed cost information in their accounting records 
to make it possible to identify the specific costs of analytic interest to the study. Therefore, we 
used direct data collection methods to obtain cost estimates specifically for the activities of 
interest in the evaluation. We did this primarily by obtaining estimates about staff time usage for 
the activities the demonstration specifically affected and by then using these staffing data, 
together with wage rate and other information, to build up estimates of the labor costs associated 
with the demonstration. We also obtained estimates of other direct costs, such as supplies, 
postage, and telephone. 

1. Data Collection 

In implementing this approach, we used the information from the interviews to design 
customized worksheets for each district that recorded the staff time and other costs of conducting 
NSLP certification and verification work. We developed separate worksheets for (1) activities 
related to receipt and processing of applications, and (2) activities related to verification 
activities.^ To provide a context for the estimates, we asked district staff what the period was in 
their district for the initial application process and what percentage of total applications they 
received during this period. In most districts, this initial period of peak activity included the first 
two months of the school year. We obtained similar information on verification activities. 

To ensure comparability across districts and to improve the chances that respondents would 
be able to provide accurate estimates, we asked respondents to focus on a particular period for 
each main function. For application activities, we asked respondents to focus on the period in 

^The worksheets also obtained information on direct certification costs, but since neither 
UFD nor GV affect direct certification activities significantly, we do not discuss those data here. 
In part, we asked about the direct certification activities to minimize the likelihood that the 
districts would confuse the work associated with direct certification with the work we were 
directly interested in. 
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which the district processed most of its applications, near the beginning of the school year. For 
most respondents, this was from August through the end of September. Our measure of output is 
the number of students approved for free or reduced-price meals by application as of October 31, 
2002. Therefore the estimate of resources used to conduct application processing pertains to 
approximately the same time period as our measure of output. 

Because of the timing of our interviews relative to verification work, we used a different 
approach to asking about verification activities, depending on whether the district was a UFD or 
GV comparison district (which performed verification activities in the fall) or a GV pilot district 
(which performed the first round in the fall and any necessary subsequent rounds in the winter 
and spring).^ We conducted the interviews from November 2002 to February 2003, and we sent 
customized cost worksheets to each respondent in January 2003. Accordingly, we asked 
comparison districts to provide the time estimate for verification activities for the current school 
year. We asked the GV pilot districts to provide this estimate for the prior school year to allow 

n 

us to capture all possible rounds of verification. Also important to note is that estimates of 
verification costs for the GV pilots depend on the number of rounds of verification each district 
was required to perform in the year in question. 

To account for the fact that staff at different levels of responsibility and pay performed the 
various activities, we placed a value on the time estimates by using the wage and salary 
information the respondent provided on each staff member or staff category involved in 

^We did not ask about verification activities in UFD pilot districts because they were not 
required to conduct verification under pilot rules. 

^In fact, two GV pilot districts provided the estimates for the prior school year; in one pilot, 
however, our respondent had not performed the verification activities in the prior year, so that 
respondent provided the information for the current school year. 
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application and verification. Initially, we had planned to obtain information on the amount of 
fringe benefits; however, our respondents were not well informed about those costs. 
Furthermore, because variations in fringe benefit costs across the 25 districts in the study would 
have introduced an additional element of chance variability instead of helping to measure more 
accurately the costs of the resources involved, we applied an average fringe benefit rate of 25 
percent to the wage and salary costs of the School Food Authority (SFA) staff who perform 
application and verification functions. 

Finally, we asked interview respondents to estimate the other direct costs in processing 
applications. We anticipated these costs would include printing, copying, postage, and the use of 
computers. 

To supplement the data collection described above, where possible, we also asked UFD 
districts for information about changes in staff requirements from the pre-pilot period to the 
period the pilot covered. In particular, we asked the SFA staff who reported on pilot procedures 
whether they had performed application and verification activities during the period just before 
implementation of the pilots. If the SFA staff reported that they had performed these tasks in 
both periods, we asked them to compare the amount of work effort during the pilot with the 
amount before it. Because of the inherent difficulties in making estimates of hours of work 
required to perform the activities involved and the three- year interval between the pre-pilot 
period and the time of our interview, we did not believe respondents would be able to report 
reliably the amount of time used to perform the various functions during the pre-pilot period. 
Therefore, we asked for their qualitative assessment of whether the work effort during the earlier 
period was greater than, the same as, or less than the work effort required under the pilot. 
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2. Cost Analysis for the UFD Pilot Projects 

After we had obtained the data described above, we used somewhat different analytical 
methods for the UFD pilots than for the GV pilots. For the UFD pilots, we base our analysis on 
directly comparing average certification costs for the pilot districts and the corresponding 
comparison districts, as of the time of the data collection. Essentially, this involves comparing 
data in the pilot and comparison districts for the 2002-2003 school year. To provide a basis for 
comparing time and resource costs of conducting application and verification activities across 
SFAs with differing numbers of applications, verifications, and approved students, we calculated 
total costs across all staff involved in a function, then expressed these costs on a per-unit-of- 
output basis. We focus our discussion on costs per approved student, a parameter that is 
consistently defined across the sites and easy to interpret. We develop estimates of the costs per 
student enrolled using information from Burghardt et al. 2004 on pilot project impacts on 
certification rates and estimates of pilot versus comparison differences in costs per approved 
student. We develop similar estimates of costs per student approved using information from 
Hulsey et al. 2004 on pilot impacts on application rates. 

From the professional estimates collected through staff interviews and worksheets, we 
calculated the total staff time by person devoted to application processing and eligibility 
determination and verification activities at each study district. We adjusted interview 
respondents’ staff time estimates for two problems. First, some respondents appeared to have 
omitted from their estimates the time associated with receiving applications in the classroom and 
transmitting them to the school food service office. Second, one respondent appeared to have 
overestimated the time required because that respondent’s estimate of the time used exceeded the 
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time available during the period that the work was performed. We adjusted the time estimates to 

o 

correct these omissions and anomalies. 

The estimates of total costs for the UFD and GV comparison districts appear to be broadly 
consistent with estimates presented in a recent report by the General Accounting Office (U.S. 
General Accounting Office 2002). While our point estimates are lower, they appear to be within 
the range that sampling error could be expected to produce. Appendix Table A. 6 presents a 
comparison of the mean costs per student approved in the comparison districts for the evaluation 
of the NSLP Application/Verification Pilot Projects with the mean costs calculated from the 
GAO report. 



3. Cost Analysis for the GV Pilot Projects 

Initially, we attempted to use the same methods of cost analysis described immediately 
above for the analysis of GV costs. The preliminary results we obtained using this method 
lacked face validity, however, which suggested that the method was not providing satisfactory 
information for this version of the pilot.^ 

As a result, we used a different analytic approach for the GV cost analysis, which drew on 
information about the pilots that allowed us to focus specifically on the costs of the activities the 



o 

Appendix Table A.l shows the adjusted time and cost estimates by pilot and comparison 
district. Appendix Table A. 2 shows the unadjusted time and cost estimates. The adjustments did 
not alter the basic findings. 

^In particular, the preliminary analysis suggested that costs were lower at the pilot sites than 
at the comparison sites, even though all the information we obtained from the executive 
interviews with district staff suggested that they should be higher, due in part to the greater 
numbers of verifications which were taking place at the GV sites. We believe that the apparently 
incorrect results from the direct comparisons of costs at the pilots compared to their comparison 
districts were due to the fact that only three pairs of sites were available for the analysis. It is 
quite possible that, by the “luck of the draw,” the three comparison sites had substantially higher 
costs than the pilot sites, even at baseline. In particular, such differences could have happened 
because certification and verification costs were not among the variables used in matching the 
comparison sites, since information on them was not available. With only three site pairs, there 
was unavoidably a substantial risk of the matching being poor on this variable. 
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demonstration affected. In particular, it was clear from the interviewing that the main change in 
procedures for GV sites was the potential addition of a second and third round of verifications. 
Therefore, we analyzed the costs associated with the GV pilots by focusing only on the 
information relating to these rounds of verification from the pilot districts to estimate the costs of 
this additional verification work. Essentially our approach relied directly on information 
provided by pilot district staff about the incremental costs of the pilot procedures, rather than on 
comparisons between pilot and comparison districts. 

We selected this approach for the GV pilots because the main additional activities — round 2 
and/or round 3 verifications — could be isolated and their costs estimated separately. On the other 
hand, staff in UFD pilot districts were usually unable to isolate the additional time required to 
perform the two work activities the pilot made necessary: (1) following up on an increased 
number of incomplete applications, and (2) making the eligibility determination using 
documentation rather than the income amounts the applicant stated on the application. Therefore, 
we did not try to make a professional estimate of the pilot- specific work activities for the UFD 
pilots. Instead, we estimated the incremental costs attributable to the UFD pilot procedures as 
the difference between pilot district costs and comparison district costs. 

The approach we used in the GV analysis assumes the costs of application processing were 
unchanged in pilot years compared to the pre-pilot period. This assumption is unlikely to be 
entirely accurate because staff needed to check for, request, and then use documentation of 
income for households whose benefits were reduced or terminated due to verification. For this 
reason, the estimates of the costs of work related to the pilots are likely to understate the costs of 
GV. We believe, however, that the cost estimates reported in Chapter IV represent a reasonable 
approximation of the true costs of the use of GV. 
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III. IMPLEMENTATION OF UP-FRONT DOCUMENTATION AND GRADUATED 

VERIFICATION 



In this chapter, we describe how the nine UFD pilot districts and three GV pilot districts 
implemented the pilot procedures for the National School Lunch Program 
Application/Verification Pilot Projects. We also discuss how the comparison districts 
implemented standard application and verification procedures. 

In the first section, we describe the procedures the comparison districts followed as staff 
conducted application and verification processing. A description of the work activities typically 
needed for application and verification processing provides a context for the rest of the 
discussion. In this section, we also describe some important dimensions of variation across 
districts in how the work was accomplished, which is helpful in understanding the variations in 
district-level costs described in the next chapter. The second and third sections of the chapter 
describe implementation of UFD and GV, respectively. In each of these two sections, we 
describe the key aspects in which operations differed from standard application and verification 
procedures, then the perceptions of staff about the pilot procedures. 

A. STANDARD APPLICATION PROCESSING AND VERIFICATION PROCEDURES 

Under NSLP regulations, students can become certified for free or reduced-price meals in 
one of two ways. First, students whose families receive food stamps, TANF, or benefits under 
the Food Distribution Program on Indian Reservations (FDPIR) can be certified for free meals 
through an exchange of information between the assistance agency and the school district. 
Certification done in this way is called “direct certification.” Under the second approach, 
families of students may submit applications on which they list all people in the household and 
provide information about the sources and amounts of income for each household member or 
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they indicate that the family receives food stamp, TANF, or FDPIR benefits. To be complete, an 
adult household member must sign the application and either provide his or her Social Security 
Number on it or indicate that the signer does not have a Social Security Number. 

Federal regulations require each SFA to verify the eligibility of a sample of approved cases 
by December 15 of each year. This is done by requesting households to provide proof of the 
amounts of income for each income source, such as pay stubs or bank account information. 

Next, we describe how the nine SFAs that served as comparison districts for the UFD pilot 
districts and the four SFAs that served as comparison districts for the GV pilot districts carried 
out the tasks associated with processing applications and conducting verifications. 

1. Description of Standard Procedures 

We describe five functions performed in administering the NSLP application and 
verification process: (1) direct certification, (2) distributing and receiving applications, (3) 

processing applications, (4) record keeping and transferring information, and (5) verification. 
All districts completed each of these functions, except, in some instances, direct certification 
(which is optional). Later in the chapter, we describe how processing differed in the pilot 
districts. 

a. Direct Certification 

In the 2002-2003 school year, five of the nine UFD comparison districts and all four of the 
GV comparison districts used direct certification (see Table III.l).^ Each comparison district 
matched its pilot district in using or not using direct certification, except for one UFD district 
pair in which the comparison district used direct certification and the pilot district did not. 



*Two of the four GV comparison districts together served as the comparison district for one 
GV pilot site. 
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KEY FEATURES OF NSLP ADMINISTRATION IN PILOT AND COMPARISON DISTRICTS 
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TABLE III. 1 {continued) 
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Staff followed one of two processes to complete direct certification. Three district pairs (six 
districts) are located in states in which, before the start of the 2002-2003 school year (during 
summer 2002), the state welfare agency sent a letter to the parents of all students in the state who 
were receiving food stamps or TANF. The letter directed families to present the letter to the 
school district if they wished to have their child certified for free meals. When a family 
presented the letter to the school district, their child was directly certified for free meals — no 
application was required. Children from families who wished to have their children directly 
certified generally provided the letter to the school district near the beginning of the school year. 

The other seven comparison districts used a matching process. The county or state agency 
administering the food stamp/TANF program gave the district a list of school-age children who 
lived in the area the district served and who were members of households receiving food 
stamp/TANF benefits. District staff then compared the list of school-age children in food 
stamp/TANF households with a district enrollment list. Children who appeared on both lists 
were directly certified for free meals. Under this approach, the matching and direct certification 
processes usually occurred during the month before school opened. 

b. Distributing and Receiving Applications 

Federal regulations require that districts operating the NSLP send to the household of each 
student enrolled in the district’s schools (1) a notice telling them about the availability of free 
and reduced-price meals, and (2) an application. All the districts in the study distributed NSLP 
applications and instructions on or before the first day of the school year, either by mailing them 

2 

The number of comparisons differs from the number of pilots, because two districts served 
as the comparison for one pilot district, and one pilot district did not use direct certification while 
its comparison district did use direct certification. 
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or by sending them home with students. Many districts distributed applications at the beginning 
of the school year in a packet that contained other important documents from the district. 
Typically, staff at the district level — often the food services director, staff in the food services 
department, or staff in the business office — prepared a master application, including updated 
income eligibility guidelines and prices for reduced-price and paid meals. In some districts, 
district staff also mailed the materials to students’ homes. However, staff at individual schools 
were often responsible for copying the materials and mailing them to families or giving them to 
teachers to distribute. 

The process for receiving applications also varied. Usually, the child returned the completed 
application to the classroom teacher near the beginning of the school year. In some districts, the 
application was to be returned to someone other than the classroom teacher, such as the cafeteria 
manager, school nurse, or school secretary. Some districts asked parents to mail the application 
to the district. 

One important way in which the application process differed was in whether a family was 
required to submit one application covering all children in the family or, alternatively, to submit 
a separate application for each child. Among the nine UFD comparison districts, six used 
multichild applications, and three required a separate application for each child in the family. 
Among the four GV comparison districts, three used multichild applications, and one required a 
separate application for each child. 

c. Processing Applications 

Processing applications involved reviewing the applications for completeness, following up 
with families who submitted incomplete applications, determining eligibility based on the 
information received, and then notifying families of their children’s meal price status. Under 
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standard program rules, the following information is required for the application to be deemed 
complete: (1) name(s), grade(s), and school(s) of the student(s); (2) a food stamp/TANF case 
number or a list of household members and total income for each member; (3) the signature and 
Social Security Number of the person filling out the application; and (4) the application date. In 
some districts, staff checked incoming applications for completeness at the time they determined 
eligibility. In others, checking for completeness was a separate step, sometimes even conducted 
by different staff. In most but not all comparison districts, staff said they contacted families who 
had submitted incomplete applications and asked them to provide the missing information. 

d. Transferring Information Within the School District 

After making eligibility determinations, district staff told their individual schools the meal 
price eligibility status of each approved student. Districts used several ways to convey this 
information. Some districts created lists of all students approved for free or reduced-price meals 
and sent the lists to their schools. Others used database systems, in which staff entered approval 
information to students’ files, after which (1) schools downloaded a roster of approved students, 
or (2) the information was automatically displayed on the computers of cafeteria workers. 

e. Verification 

Standard verification in the NSLP requires one round of verification of a sample of 
approved applications. Districts first select samples of approved applications. Most comparison 
districts used random sampling, in which a sample of three percent of approved applications is 
selected. Some districts used focused sampling, in which verification samples are generally 
smaller and target households whose incomes are close to the eligibility threshold. District staff 
send a letter to each selected household notifying them that they have been selected for 
verification and stating that the family must provide documentation of its income for the most 
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recently completed month or its FS/TANF/FDPIR participation. When the family submits 
documentation, the district calculates meal price status based on information provided at the time 
of verification and adjusts the meal price status, if necessary. If a household does not supply 
documentation as specified in the letter, the district is supposed to terminate the free and 
reduced-price meal benefits of the students in the family. Verification is required to be 
completed by December 15 of each year. 

All nine UFD comparison districts used random sampling, while three of the four GV 
comparison districts used focused sampling. The number of cases that comparison districts 
reported verifying ranged from 3 to 70. The mean number verified was 22, and the median 
was 12. 

2. Variations in Staff Involved and Use of Technology 

We have described the basic procedures that study districts used to complete NSLP 
application and verification work. Before discussing the elements of application and verification 
that the pilot procedures directly affected, we describe variations in the categories of staff 
involved and in the use of technology. 

a. Types of Staffing and Degree of Centralization 

The types of staff responsible for conducting NSLP application and verification work varied 
considerably. In most districts, activities were largely centralized at the district level. In some 
districts, however, school-level staff performed at least some tasks. In smaller districts, one 
person often did all the certification and verification, but larger districts typically involved more 
than one staff person. 

In II of the 13 comparison districts in our study, NSLP eligibility determination and 
verification work was centralized and performed by staff working at the district level. Teachers 
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or other school staff distributed applications to students, received completed ones, or checked 
applications for completeness; but district-level staff completed all application processing and 
verification work. These staff members included food service directors and their field managers, 
assistants, and secretaries, or staff in the business administrator’s office or the superintendent’s 
office. In one comparison district, all of this work was performed at the school level; in another, 
school-level staff performed eligibility determination, while district-level staff conducted 
verification. 

b. Use of Technology 

Most districts used computers to perform at least some tasks, including performing direct 
certification (matching state lists of children on food stamps/TANF with enrollment records), 
making eligibility determinations, maintaining student records, transmitting certified results to 
schools, and selecting the verification sample. However, other districts performed some or all of 
this work manually. 

Differences in computer capabilities and usage could have affected the amount of time that 
staff spent completing NSLP tasks, thereby creating variations in labor costs across districts. 
Although our cost estimates include some non-labor categories such as postage, paper, printing, 
and telephone calls, they probably do not capture the full costs of computer equipment and its 
support. Thus, the cost estimates take into account the time savings gained by using computers, 
but they omit the costs of purchasing and maintaining the computer systems. 



-5 

Districts have different accounting practices with regard to how investments in computer 
systems are amortized. It was not practical to derive accurate estimates of the annual computer 
costs associated specifically with certification and verification activities. 
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B. UP-FRONT DOCUMENTATION 



Based on an examination of the reports the UFD districts submitted to USDA and on MPR’s 
interviews of district staff, we conclude that all the UFD districts operated their pilot procedures 
in basic conformance with USDA intentions and with the proposals the districts submitted. 
Here, we describe the operational changes doing so required. 

UFD altered two aspects of eligibility determination and verification: (1) it required that all 
households submit, with their applications, documentation of all household income, or, if a case 
number was provided, proof that the household was receiving food stamp, TANF, or FDPIR 
benefits; and (2) it eliminated the usual NSLP sample-based verification to be completed by 
December 15. However, the UFD pilot procedures did not alter other processes, including direct 
certification, distributing applications, or record keeping and transferring information to 
individual schools and cafeterias. 

The two most important operational changes involved the process of reviewing applications 
for completeness and making the eligibility determination. Under the pilot procedures, more was 
required for the application to be complete: rather than the district accepting the applicant’s 
statement about income amounts, the applicant was required to provide documentation showing 
gross income for each source reported. If the applicant reported a source of income but did not 
provide documentation showing the amount of income from that source, the district was required 
to consider the application incomplete unless the household reported that documentation was 
unavailable. 

Not surprisingly, the percentage of applications that were initially incomplete was higher in 
UFD districts than in their comparison districts. Based on information reported to USDA by 
staff at UFD pilot districts, the percentage of applications initially incomplete in school year 
2002-2003 ranged from 1 to 63 percent, with a median of 42 percent. Fully comparable data 
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were not available for the UFD comparison districts. However, interview respondents in seven 
of nine UFD comparison districts were able to provide estimates of the percentage of 
applications that were initially incomplete, and these estimates ranged from 1 to 20 percent, with 
a median of 10 percent. Income verification documents were the items most commonly missing 
on UFD pilot district applications. 

All respondents in UFD pilot districts reported that SFA or school personnel followed up 
with fa mi lies that submitted incomplete applications. Nearly all sent the incomplete application 
back to the household with a note indicating what was missing or sent a letter. Many also 
telephoned the household to request the information. Most comparison district respondents also 
indicated that they followed up with households that submitted incomplete applications. 

Data the UFD districts provided to the Food and Nutrition Service show that, on average 
across the districts, 92 percent of initially submitted applications were eventually deemed 
complete and either approved or denied."^ 

We asked SFA staff who implemented UFD procedures for their perceptions about the pilot 
procedures. As we report next, they noted both advantages and disadvantages. 

1. Advantages of UFD 

In general, staff responded that the main advantages of UFD were increased accuracy of 
application information and increased fairness to families. These responses reflected concerns 
that staff seemed to have had with the standard procedures before the pilots. In particular, staff 
in some districts expressed concern about the accuracy of the income information families 
provided on NSLP applications under standard procedures. They felt that accuracy was 

'^Memorandum from Paul Strasberg, FNS Project Officer to Mathematica Policy Research, 
Inc., dated February 25, 2003. 
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sometimes compromised, for at least two reasons. First, they suspected that families do not 
necessarily understand the difference between gross and net income, so some households report 
the income they receive, rather than their total income before taxes and other deductions. 
Second, some staff said that families have expressed confusion and disagreement with the use of 
the income of all household members to determine eligibility for the NSLP, which can result in 
some families reporting only the incomes of parents or heads of household and exclude incomes 
of other household members. Staff suggested that families may misreport income information on 
their NSLP applications for reasons such as these, either by accident or intentionally. Under 
standard procedures, staff have no way of confirming that the stated income is the family’s gross 
income when they are processing the application; the information is checked only for those 
applications selected for verification. Many staff in UFD districts felt that their procedures 
increased the accuracy of the application process. Some staff also believed UFD was fairer to 
families. One staff member said that, before the pilot, many families selected for verification felt 
that the district did not trust them and that they were being singled out, since other families were 
not being asked to provide income documentation. She preferred UFD procedures because all 
families were treated the same. 

2. Disadvantages of UFD 

Disadvantages of the UFD pilots that pilot district staff reported included the high levels of 
incomplete applications received and complaints from families who did not want to provide 
income documentation. The most important disadvantage of UFD, mentioned during several of 
our interviews with UFD districts, appears to be the number of incomplete applications. In 
particular, many families in UFD districts submitted NSLP applications without the required 
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income verification documents.^ Contacting families to request additional information required 
extra time and effort from staff. However, some staff noted that the first year of the pilot was the 
most difficult and that the number of incomplete applications had decreased as families became 
accustomed to the pilot procedures. 

C. GRADUATED VERIFICATION 

Information provided in the GV districts’ reports to USDA and by the interviews with 
school officials conducted for the evaluation suggests that the implementation of the GV 
alternative was less consistent and complete than the implementation of the UFD model. Three 
of the four original GV pilot districts did not implement the model completely for all three 
school years of the demonstration. One GV pilot district was excluded from this evaluation 
because of difficulties in performing later rounds of verification during the first and second year 
of the pilot and failure to require documentation at application for students whose benefits were 
reduced or terminated due to verification the previous year. A second district, which was 
included, did not perform the required up-front documentation at the beginning of each new 
school year of households whose benefits were reduced or terminated in the previous school year 
due to verification. The third withdrew from the pilot project in the third year because of 
perceived adverse effects on the number of students certified for free or reduced-price meals and 
therefore on its allotment of state and federal funding. Next, we describe how program 
procedures were changed when the pilot procedures were being used. 

GV entailed changes in two aspects of application processing and verification. First, it 
could increase the number of applications subject to verification. One key component of the GV 

^Some staff suspected that families did not, or could not, read the full application materials, 
noting that some enrolled families struggle with illiteracy and limited English proficiency. 
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pilot was that, if as a result of the standard verification process more than 25 percent of cases had 
benefits reduced or terminated, then the SFA conducted verification on half of the remaining 
applications in a second round of verifications. If 25 percent of more of the applications verified 
in the second round resulted in benefit reduction or termination, the district verified all the 
remaining cases in a third round of verifications. Table III. 2 shows the number of rounds of 
verification by year in each of the three GV pilot districts included in the evaluation. As shown, 
the number of rounds of verification varied by year. In each of the first two years of operation, 
most GV districts conducted two or three rounds of verification. In year three, the two districts 
continuing in the demonstration did just one or two rounds of verification. 

The second main feature of program administration that changed in the GV pilot districts 
was that households whose benefits were reduced or terminated due to verification in a given 
school year were required to submit documentation of their income if they reapplied during the 
current school year or applied during the school year following the one in which benefits were 
reduced or terminated. 

In school year 2002-2003, two of the three GV pilot districts included in the evaluation 
implemented this requirement for those whose free and reduced-price meal benefits had been 
reduced or terminated in 2001-2002 due to verification. In one district, the requirement was 
implemented by manually checking each application in the current school year against a list of 
students whose benefits were reduced or terminated in the prior school year. In the other district, 
such cases were flagged automatically as their new application information was entered to an 
automated system for determining eligibility. In both districts, cases that were reduced or 
terminated in the prior school year were checked to determine whether the household had 
provided documentation of income. If income documentation was not provided with the 
application, a letter was sent to the household requesting the documentation, and the application 
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NUMBER OF ROUNDS OF VERIFICATION IN GRADUATED VERIFICATION PIEOT DISTRICTS, 

BY DISTRICT AND YEAR 
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was held as incomplete until the documentation arrived. In the third GV pilot district, the staff 
responsible for determining eligibility were unaware of the requirement, and the requirement was 
not implemented. 

None of the staff in the three GV districts expressed positive feelings about the pilot, and 
none named any advantages of the pilot procedures. They expressed frustration both at the level 
of nonresponse among families to verification and at the amount of work required to perform the 
additional rounds of verification. Not only did the additional rounds constitute extra work for 
staff, but the workload was difficult to plan for. Staff did not know until they received the results 
of the previous round of verification whether they were finished or would have to undertake 
another round. Furthermore, the volume of work associated with the second and third round was 
considerably greater than the volume of work for the first round. Staff in all three GV districts 
felt strongly that completing the extra rounds of verification required more time and work than 
they had expected. All agreed that the extra verification work required too much time to 
complete. One staff person noted that the pilot more than tripled the amount of work usually 
required to perform verification. 

Many families did not respond to the request for verification documents in GV districts. 
One district estimated the level of nonresponse at 50 percent; another estimated it to be about 40 
percent.^ Furthermore, because GV pilot districts conduct up to three rounds of verification (and 
verify up to 100 percent of their approved students), the percentage estimates of nonresponse in 
these districts imply a much larger number of families verified than in comparison districts. 



^Only one comparison district gave an estimate of the level of nonresponse to verification 
under standard procedures. It believed it was about a third of the students selected. 
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IV. IMPACTS ON COSTS 



In this chapter, we examine the costs of using UFD and GV pilot procedures. We address 
the following question: What were the additional costs or cost savings due to the changes in 
procedures implemented to carry out the National School Lunch Program 
Application/Verification Pilot Projects? 

As described in Chapter III, UFD required that SFA staff obtain and review documentation 
of income for all applicants at the point of application. However, no further verification was 
performed later in the school year. In GV districts, SFA staff increased the number of cases 
verified if the proportion of cases with a benefit reduced or terminated exceeded 25 percent in a 
previous round of verification. 

We estimated the impact on cost per approved student for UFD districts using the first 
method described in Chapter II — contemporaneous comparisons between the pilot districts and 
their matched comparison sites — drawing on professional estimates by district staff of the 
resources needed to implement the certification and verification procedures. We estimated the 
impact on costs per approved student for the GV districts using professional estimates by the 
staff involved of the resources needed for the additional verification work. We then estimated 
impacts on overall costs by multiplying our estimates of cost per approved student with estimates 
of the proportion of all students approved for free or reduced-price meals in pilot and comparison 
districts. 

A summary of the findings is as follows: UFD caused relatively modest increases in the 
costs of eligibility determination per student approved by application, and GV caused increases 
in costs that were somewhat larger in relative terms. We have a great deal of confidence in these 
qualitative findings. However, the point estimates presented are subject to considerable 
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uncertainty, both because the number of districts is small and thus subject to sampling variability 
and because the individual professional estimates on which the district estimates are based are 
subject to considerable measurement error. In UFD districts a reduction in the percentage of 
students certified offset the increase in costs per student approved such that total costs of 
eligibility determination were unchanged. In GV districts, although a reduction in the percentage 
of students certified partially offsets the increase in cost per approval, total costs of eligibility 
determination increased. 

A. IMPACTS ON COSTS PER APPROVED STUDENT IN UFD PIUOT SITES 

Table IV. 1 presents a summary of key data for UFD pilot and comparison districts. Across 
the nine UFD pilots, staff spent, on average, approximately 22 minutes per approved student to 
complete all the work associated with application processing. In the UFD comparison sites, staff 
spent approximately 19 minutes per approved student completing application processing and 
verifications. Thus, pilot district staff spent just under three minutes more per approved student 
in application processing and verification. When we take into account salary costs and other 
costs associated with application processing and verification, the average total cost is $7.71 per 
approved student in UFD pilot districts and $6.21 per approved student in UFD comparison 
districts, an impact of about $1.50 per approved student. 

These estimates are subject to considerable margins of error, because of the very small 
number of pairs of districts available for observation. Indeed, this is highlighted by the findings 
for the individual district pairs (see Appendix Table A. 1). In six of nine pilot-comparison district 
pairs, the time and cost estimates were greater in the pilot district than in the comparison district. 
In three district pairs, however, time and costs were estimated to be higher in the comparison 
district. It is extremely unlikely that costs could have been lowered by the implementation of 
UFD in any of the pilot districts, given what we know about how the pilots operated. Thus, these 
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TABLE IV. 1 



UP-FRONT DOCUMENTATION: STAFF TIME AND TOTAF COSTS FOR 
APPFICATIONS AND VERIFICATION PIFOT AND COMPARISON SITES 





Pilot Districts 


Comparison Districts 


Difference 


Staff Minutes per 
Approved Student 


21.6 minutes 


19.1 minutes 


-1-2.5 minutes 


Wage Costs Plus 
Other Costs per 
Approved StudenC 


$7.71 


$6.21 


-t$1.50 



^Based on (I) applying staff wage information and information about fringe cost to the labor time 
data shown in the table, and (2) tabulating data on reported non-labor costs. 
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three pairs with negative results probably reflect poor initial matching of pilot and comparison 
districts on this variable. We have retained them in the analysis, however, since there may also 
be sampling error in the other six pairs that could potentially inflate the cost estimates. Overall, 
these estimates indicate that implementation of UFD increased staff time per approved student 
by about 13 percent and total cost per approved student by 24 percent.^ 

Evidence from the implementation study broadly supports the estimated cost increases 
reported in Table IV. 1. First, in general, the staff we interviewed reported that integrating into 
their application processing routine the additional work of checking applications for 
documentation, contacting families who had not provided documentation, and reviewing 
documentation when they made eligibility determination was not unduly burdensome. Second, 
we asked the five staff members interviewed in UFD pilot districts who had performed 
application processing and verification both before and during the pilot period to assess the 
change in work effort that the pilot procedures caused, and all five reported that the pilot had 
created, at most, a modest amount of additional work for them. 

B. THE IMPACT ON COSTS PER APPROVED STUDENT IN THE GRADUATED 

VERIFICATION PILOT DISTRICTS 

Table IV. 2 shows our estimate of the pilot project’s impacts on costs in the GV pilot 
districts. To provide a basis for assessing the relative sizes of changes in costs due to the pilots, 
the first column in the table displays the mean across the three GV pilot districts of the 
professional estimate of staff time and total costs for application processing and the first round of 
verification. Based on the information learned from interviews with staff, it appears that the 
demonstration did not significantly affect these costs. 



'Calculated as difference* 100 divided by mean value in comparison districts: 
2.5%*100/19.1% = 13.1% and 1.50%*100/6.21% = 24.2%. 
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TABLE IV.2 



GRADUATED VERIFICATION PILOTS: STAFF TIME AND TOTAL COSTS FOR 
APPLICATION PROCESSING AND FIRST ROUND OF VERIFICATION AND DECREMENT AL 
COSTS FOR ADDITIONAL ROUNDS OF VERIFICATION 





Base Resources for Application 
Processing and First Round of 
Verification (Not Substantially 
Affected by the Demonstration) 


Incremental Costs for Second And 
Third Rounds of Verification 
(Estimate of Increased Costs Due to 
the Demonstration) 


Staff Minutes per 
Approved Student 


9.1 minutes 


6.6 minutes 


Wage Costs Plus Other 
Costs per Approved 
Student 


$3.75 


$3.06 
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The second column shows the estimates for the second and third rounds of verification. 



Because the staff interviews were conducted in December 2002 and January 2003, before all 
rounds of verification had been completed, and because one district was no longer participating 

in the pilot in the 2002-2003 school year, we asked staff to estimate the time required to perform 

2 

verification activities in the 2001-2002 school year. 

The estimates indicate that staff time for taking applications and the first round of 
verifications was about nine minutes per approved student, and the total costs for this was $3.75. 
Staff time for the second and third rounds of verification was 6.6 minutes per approved student, 
and total costs were about $3 per approved student. This comparison indicates that GV 
substantially increased the amount of staff time and the costs associated with application 
processing and verification.^ 

Two points are important for assessing these estimates. First, the amount of additional work 
and costs depends greatly on the number of rounds of verification that must be performed. 
During the 2001-2002 school year, one GV pilot district conducted two rounds of verification, 
and two districts conducted three rounds. Thus, we observed eight verification rounds in the 
observation year chosen across the three GV pilot districts. Since the possible total number of 
rounds of verification is between three and nine, the number of rounds actually observed is close 
to the maximum number that could have been observed. Accordingly, setting aside problems in 



2 

Respondents in two of the three districts were not able to provide such estimates, either 
because they could not recall how long the work took or they did not perform the relevant tasks 
in the previous school year. In these situations, we requested estimates for the current school 
year and used these to develop estimates for the rounds of verification that our respondents were 
not able to report on. 

-5 

Table A. 2 provides estimates of the measures in Table IV. 2 for the three GV pilot districts. 
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measuring staff time and costs accurately, the additional amount of work due to the pilot 
procedures that was actually observed was near the maximum possible increase. 

Second, the professional estimates that staff at the three GV pilot sites provided tended to 
place these districts in the lower part of the distribution across all districts in the amount of staff 
time required for application processing and conducting verification. In contrast, as noted 
earlier, the GV comparison districts tended to provide much higher estimates (see Table A. 3). 
We cannot determine the extent to which such variation is due to differing levels of accuracy in 
the professional estimates and the extent to which it reflects real differences in the way work is 
performed. As noted in Chapter II, however, this leads us to conclude that the comparison 
between the pilot sites and comparison sites, which is the basis of our estimate for UFD, was not 
giving us an accurate estimate of the differences in costs. The approach we have taken gets 
around this problem by essentially having each pilot district serve as its own comparison. 

In all three GV pilot districts, we were able to interview a person who had been involved 
with application processing and verification work both before and during the pilots. They 
unanimously reported that the additional verification work was substantial and quite burdensome 
because of its unpredictability and extra resources needed to increase the overall number of 
verifications. Thus, as in the UFD analysis, our quantitative estimates of the costs of GV are 
broadly consistent with the reports of pilot staff concerning their perceptions of the burdens and 
costs of the pilot procedures, as reported in Chapter III. 

C. IMPACTS ON APPLICATION AND VERIFICATION COSTS PER STUDENT 

ENROLLED AND PER STUDENT APPLYING 

In this section we estimate the change in total costs of processing applications and 
conducting verifications that was caused by the UFD and GV pilot projects. In conducting our 
cost analysis, we first estimated total costs at each study district and then normalized on three 
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different measures of the size of the district: number of students enrolled, number of students 



applying for free and reduced-price meals, and number certified for free and reduced-price 
meals. Below, we present estimates of the changes in costs per enrolled student, as well as the 
change in cost per student applying for free or reduced-price meals and the change in cost per 
student approved for free and reduced-price meals. 

1. Costs per Enrolled Student in UFD and GV Districts 

Analysis presented in the previous sections indicates that UFD increased the costs of 
application processing and verification by about $1.50 per approved student, or by about 25 
percent. Furthermore, GV increased these same costs by about $3.00 per approved student, 
which was an 80 percent increase over the estimated base costs of processing applications and 
conducting verifications in the GV districts. Estimating the change in total costs requires that we 
incorporate information about the effects of the demonstrations on the percentage of students 
approved for free or reduced-price school meal benefits. 

For each district group, we estimate the average total cost per student enrolled as the average 
cost for application and verifications per student approved for free or reduced price meals in the 
district group, multiplied by the proportion of students approved. We estimate the impact on 
costs as the difference in total costs in pilot districts and in the counterfactual situation. 

As Table IV. 3 shows, the average cost for application and verifications per student enrolled 
is $1.34 in pilot districts and $1.25 in comparison districts, for a net additional cost of $.09. The 
difference is about 7 percent of the average comparison district cost. The percentage difference 
in cost per enrolled student is smaller than the percentage difference in cost per approved 
student, because the pilot procedures reduced the proportion of all students approved for free or 
reduced-price meals. Furthermore, as reported in Burghardt et al. 2004, all of the reduction in 
certification was due to the fact that a smaller proportion of students from income-eligible 
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COSTS FOR APPLICATION AND VERIFICATION PER STUDENT APPLYING 
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districts. Thus, we use our estimate of base costs as an estimate of comparison district costs. 




families were approved in the UFD pilot districts than in UFD comparison districts. The 
demonstration did not have a measurable effect on free and reduced-price approval rates among 
students from families with incomes exceeding 185 percent FPL. 

The cost per enrolled student was $2.10 in the GV pilot districts and $1.35 in the GV 
comparison districts, for a difference of $.75."^ This difference is 56 percent of the base cost per 
student for application and first round verifications. In percentage terms the difference in cost per 
student enrolled is somewhat smaller (at 56 percent) than the difference in cost per approved 
student (at 82 percent). This occurs because the savings that results from lower certification 
rates only partially offsets the additional costs that are due to the higher costs per approved 
student. 

2. Cost per Student Applying and per Student Approval in UFD and GV Districts 

We estimate the costs of the pilot project on the cost per student applying using the same 
basic method as the one used to estimate cost per student enrolled. We multiply the cost per 
student approved by the proportion of students applying who are approved. The proportion of 
students applying who are approved is estimated as the proportion of students approved divided 
by the proportion of students applying. 

Table IV. 3 provides data used in the calculation and the next to last row shows the cost per 
student applying. Because relatively high percentages of students applying are approved, the cost 
per student applying is similar to but less than the cost per student approved. The cost per 
student applying was $6.48 in UFD pilot districts compared to $5.68 in UFD comparison 

"^For this calculation we use the regression-adjusted estimated approval rates in GV pilot and 
comparison districts, and we used the cost per approved student for the costs of the base 
resources to conduct application processing and first round verifications as the comparison 
estimate in this calculation and the base resource cost plus incremental costs of the second and 
third round of verifications as the pilot district estimate. 
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districts, a difference of $.80. The increase in cost per student applying due to the UFD pilot 
procedures was 14 percent. This difference is smaller in percentage terms than the difference in 
cost per approved student because a lower approval rate in the UFD pilot districts partially 
offsets the higher cost per approved student in the UFD pilot districts.^ 

In GV districts the cost per student applying was $6.06 in the pilot districts and $3.51 for the 
counterfactual, a difference of $2.55. The difference in cost is 73 percent of the base costs, very 
similar to the 80 percent difference in costs per student approved. 



^Using data presented in Hulsey et al. 2004, Table II.2, we calculate that the percentage of 
applicants approved was lower in the UFD pilot districts both for students from families with 
income less than 185 percent of the federal poverty level (88 percent vs. 93 percent in 
comparison districts), and for students from fa mi lies with income greater than 185 percent of the 
federal poverty level (72 percent versus 76 percent in comparison districts). 
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V. SUMMARY OF FINDINGS 



This chapter summarizes the findings of our study of pilot operations and costs. Here, we 
offer our assessment of the extent to which the pilot projects represented a fair test of the basic 
model that the Food and Nutrition Service envisioned. The chapter also highlights some key 
advantages and challenges in carrying out the procedures necessitated by the pilot projects. 

A. UP-FRONT DOCUMENTATION 

The UFD pilot project involved two changes of standard NSLP application and verification 
procedures: (1) it required that all applicants submit documentation of all income sources with 
their application, and (2) it eliminated the verification of income for a sample of households 
conducted in late fall. All the pilot districts implemented the pilot as planned. Some pilot 
district staff believed that it improved the accuracy of income reporting, largely by relieving 
families of the need to distinguish between gross income and net income — families could simply 
provide pay stubs from which school staff could determine gross wages. The process was also 
perceived by district staff as fairer because all families, rather than a small sample, were subject 
to the documentation requirement. 

Several limitations offset these advantages. Pilot districts received more initial incomplete 
applications, so staff had to follow up to secure all the documentation. Data presented in Chapter 
ITT are consistent with the proposition that a larger proportion of initial applications in the UFD 
pilot districts were never completed, although limitations of the available data do not allow us to 
reach this conclusion with confidence. A second limitation that staff at several pilot sites pointed 
out is that the documentation requirement did not prevent families who wanted to conceal some 
of their income from reporting and documenting some income sources while failing to report 
others. 



43 




Pilot project staff felt that UFD created some additional work but that this increase in work 
was manageable and not overwhelming. Furthermore, our formal cost estimates in Chapter IV 
confirm that UFD creates at least a modest increase in application-processing costs per approved 
student. Because the UFD pilot procedures reduced the percentage of students who were 
approved for free and reduced-price meals, total costs of eligibility determination were 
unchanged by the UFD pilot procedures. However, this cost neutrality came at the expense of 
targeting efficiency because all of the reduction in certification was due to reduced certification 
rates among eligible children. 

B. GRADUATED VERIFICATION 

Compared to the UFD model, district staff felt GV was more complex to operate and more 
burdensome for staff to implement. In addition, the logic of GV’s objectives was not as clear to 
the staff responsible for implementation. GV was complex because it required one, two, or three 
rounds of verification, depending on whether the percentage of cases reduced or terminated at 
each stage was above 25 percent. A result of this design was that the workload, especially the 
large workload associated with the second and third rounds of verification, was not easy for SFA 
staff to predict or plan for. Staff had to find time in already tight schedules to carry out these 
later rounds, if they proved necessary. 

GV was also more complex at the application stage. This was because most families were 
subject to the standard verification procedures, but some were subject to the same process as all 
families followed in the UFD pilots. Furthermore, the number of households required to provide 
documentation varied from year to year and depended heavily on the number of rounds of 
verification conducted in the prior school year. Careful record keeping was necessary to support 
this determination. 
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Because of these complexities, GV was generally not implemented with same degree of 
fidelity to the original pilot model as was the simpler UFD model. Indeed, three of the four 
original GV pilot districts did not implement the model completely for the three school years of 
the demonstration. 

Our analysis of costs confirmed that GV increased the cost of eligibility determination per 
approved student. While the GV pilot reduced the number of students approved for free and 
reduced-price meals, the increase in cost per approved student was such that the overall cost of 
eligibility determination per enrolled student increased by about $2.55 — a 73 percent increase 
over the base costs. 
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APPENDIX A 

SUPPLEMENTARY TABLES 
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Estimate includes other direct costs in addition to wage costs for applications and verifications. Date are adjusted tocorrect apparent anomalies and omissions, 
by district. 




TABLE A.2 



GRADUATED VERIEICATION PILOTS: RESOURCES PER APPROVED STUDENT FOR APPLICATION PROCESSING 
AND FIRST ROUND OF VERIFICATION AND INCREMENTAL RESOURCES FOR ADDITIONAL ROUNDS OF 

VERIFICATION, BY DISTRICT 





Minutes per Approved 
Student for Application 
Processing and First 
Round of Verification 
(Not Substantially 
Affected by the 
Demonstration) 


Incremental Minutes 
per Approved Student 
for Second and Third 
Rounds of Verification 
(Estimate of Increased 
Costs Due to the 
Demonstration) 


Total Costs per 
Approved Student for 
Application Processing 
and First Round of 
Verification (Not 
Substantially Affected 
by the Demonstration) 


Incremental Total 
Costs per Approved 
Student for Second and 
Third Rounds of 
Verification (Estimate 
of Increased Costs Due 
to the Demonstration) 


GV District I 


5.7 


2.2 


$2.78 


$1.32 


GV District 2 


17.6 


15.2 


$7.33 


$6.32 


GV District 3 


3.9 


2.6 


$1.13 


$1.53 


Mean 


9.1 


6.6 


$3.75 


$3.06 
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ADJUSTED STAFF TIME AND OTHER COSTS FOR APPLICATIONS AND VERIFICATIONS PER APPLICATION, BY DISTRICT 
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Estimate includes other direct costs in addition to wage costs for applications and verifications. 




ADJUSTED STAFF TIME AND OTHER COSTS FOR APPLICATIONS AND VERIFICATIONS PER STUDENT ENROLLED, BY DISTRICT 
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Estimate includes other direct costs in addition to wage costs for applications and verifications. 




TABLE A.6 



COMPARISON OF COST ESTIMATES FROM THE COMPARISON SITES IN THE 
EVAEUATION OF THE NSEP APPEICATION/VERIFICATION PIEOT PROJECTS 
AND GAO’S STUDY OF SCHOOE MEAE PROGRAM COSTS 





Pilot Evaluation 
Comparison Sites^’ 


GAO Study'’ 


Number of Districts 


13 


10 


Mean Costs per Student Approved for 






Free and Reduced-Price Meals 


7.51 


9.84 


95-Percent Confidence Interval 


(5.36, 9.65) 


(5.69, 13.99) 



^Estimates pertain to the 13 pilot evaluation comparison districts. Cost figures include costs of 
processing applications and conducting verifications. Costs of performing activities related to 
direct certification are excluded. Mean cost in the pilot evaluation comparison districts differs 
from the simple mean of the comparison district means shown in Table A. 1 (column displaying 
“Wage Costs plus Other Costs for Applications and Verifications, per Approved Student”). The 
reason for this difference is that here we include as separate observations both comparison 
districts for the one pilot district in which the comparison sample was selected from two 
neighboring districts, whereas in Table A.l we include the mean value of the two districts as the 
value for the comparison to that pilot district. See Burghardt et al. 2004 for discussion of the 
comparison districts. 

'’Estimates are calculated from data presented in Tables 3, 5, and 7 in “School Meal Programs: 
Estimated Costs for Three Administrative Processes at Selected Eocations,” U.S. General 
Accounting Office, September 2002. Based on description in the GAO report, we believe the 
GAO estimates are developed using professional estimates. These estimates very likely include 
the costs of direct certification, which are not included in our estimates. 
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APPENDIX B 

PROTOCOLS FOR COLLECTION OF PROCESS AND COST DATA 




NSLP APPLICATIONA^ERIFICATION PILOT EVALUATION 

PROCESS STUDY 



Protocol for Interviews with District and School Administrators and Staff 



I. CURRENT PROCEDURES 
A. Direct Certification 

1. Does your school district use direct certification to certify students whose families are 
receiving food stamps and/or TANF benefits? 

2. How is direct certification conducted in your district? (does the district receive a list of 
students receiving FS/TANF, or does the welfare agency send letters to families receiving 
FS/TANF that they can bring to the school district?) 

a. Do students (or families) have to return letters to the district in order to become 
directly certified? 

3. Can you list the steps or tasks that the district performs in order to conduct direct 
certification? 



4. Do schools perform any work for direct certification? What steps/tasks? 

5. Which staff are involved in conducting direct certification. . . 
a. . . .at the district level? 



b. . . .at the school level? 



6. When does most of the work conducted for direct certification occur? Is there a deadline 
for completing direct certification? 

7. Can students be directly certified throughout the school year? Under what 
circumstances? 



8. How much (if any) work occurs after the main period (at the beginning of the year) for 
additional certifications? 



59 




9. What records and/or data are kept concerning students who are directly certified? Are 
these kept at the district or school level, or both? 



10. What happens if a family who has been directly certified submits a written application? 



1 1 . a) What percentage of children approved to receive free meals in the district are 
approved on the basis of direct certification? 

b) What percentage of children are approved on the basis of an application? 



B. Application Procedures (Written Applications) 

Outreach and Dissemintion: 

12. Are applications completed for each student or per household? 

a. Would it be possible for us to see a copy of an application form? Send/fax/email 
to us. 



13. How are households informed about the NSLP and the availability of free and reduced 
price meals? 

14. Who is responsible for this outreach (at what level)? 

15. When does outreach occur? Does it take place just once, or at different times during the 
school year? 

16. Who is responsible for actually disseminating applications (at what level)? 

17. How is this done (given to students vs. mailing them to homes, etc.)? 

18. Which children/families do you distribute applications to? 

19. How do you ensure that all households are reached? 
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20. (GV Pilot Only) How is it communicated to families whose NSLP benefits were reduced 
or terminated in the prior year that they must submit income documentation with their 
application? What process does the district use to flag these students, and to make sure 
that these children and their families are alerted that they must submit documentation? 



21. When are applications distributed? Does this happen once or more during the year? 



Receiving Applications; 

22. How are incoming applications received (via mail, turned in by students, etc.)? 

23. Who receives the applications initially, and what is done with them? (Are the staff 
involved in receiving applications located at the district or school level?) 

24. When are applications received? Is there a deadline for receiving applications in order to 
be reviewed in the beginning of the school year? 

25. Does the district accept new applications throughout the school year? Under what 
circumstances? 



Reviewing Applications; 

26. What is the process used to review applications? 

27. Which staff perform this review? 

28. At what point during the process of receiving/reviewing applications are applications 
checked for completeness (including proper documentation of household income for 
Alternative 1)? 

29. (UFD Pilot or GV Pilot for families who’s benefits were reduced last year) What 
documentation is generally provided by families with their applications? 

a. What documentation (if any) is required by students who are categorically eligible for 
free meals? 



30. What constitutes a complete application? 
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3 1 . Approximately what percentage of applications are complete and include proper 
documentation when they are initially submitted? 

32. Approximately what percentage of applications are never completed with proper 
documentation? 



33. How do you handle incomplete applications? 

34. (GV Pilot only) How are staff who review applications made aware of families who must 
submit income documentation at the time of application due to a reduction or termination 
of benefits in the prior year? 



Eligibility Determination and Notification 

35. What information from the application is used to determine eligibility for free or reduced 
price meals? 



36. (UFD Pilot only) How is the income documentation used in order to determine 
eligibility? 

37. When are applications reviewed and eligibility determinations made, for families who 
submit complete applications initially? 

38. How are families notified about eligibility decisions? 

39. How are appeals handled? 



Submitting Additional Information: 

40. How are families who submitted incomplete applications notified that additional 
information/documentation is needed? 



41. Is there a deadline for submitting additional information? If so, when is this, generally? 



42. How are re-submitted applications received and reviewed (using what criteria)? 
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43. Which staff are involved? (Are the staff located at the district or school level) 



44. During what time period does this work generally take place? 



45. Under what circumstances do families submit information/documentation regarding 
changes in household circumstances during the year? How often does this occur, and 
what happens when it occurs? 



Record Keeping and Information Transfer: 

46. Where and in what media are records about families’ eligibility maintained? 



47. What information do the records contain? 



48. Are applications submitted for each student, or for each household, or some combination? 



49. How does information about students’ eligibility move from the district level to 
individual schools? What staff are involved at each level? 



50. How is this information made available to staff working the cash register in the cafeteria? 



51. If files of applications are kept, ask to review a few (selected at random). 



General: 

52. We’ve discussed the steps involved in disseminating, receiving, and reviewing 
applications, and notifying families of their eligibility status for the NSLP. Are there 
additional tasks that we have not talked about, that are involved in the application 
procedure for free and reduced price meals in your district? (If so, list activities, staff, 
level, and time frame) 

53. What is the time period during which most of the work on applications takes place — from 
disseminating the applications to determining eligibility and transferring that information 
to the schools? 

54. In 2002, how many applications did you receive during this time period (specified 
above)? 

55. How many of those applications were approved (F/RP separately if possible)? 
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56. And how many applications were received in the months following that time period? 
(info from 2001-2002 year is okay) 

57. How many F/RP approvals were made during those months? 

58. To what extent does nature and/or the amount of work done on applications vary from the 
initial period in the beginning of the school year, to the months afterwards? 



B. Verification (GV Pilot and Comparison Districts Only) 

Sample Selection: 

59. How is the initial verification sample selected? 

60. (for districts where applications contain multiple children): Is the sample selected 
using students, or using applications? 

61. Who performs this task? At what level (district or school)? 

62. When during the school year is the sample selected? 

63. How many applications/students were selected and verified during (the first round of) the 
2001-2002 school year? 



Requesting and Receiving Documentation: 

64. How are families notified that they have been selected for verification? 



a. Could you send or fax to us an example of the notification letter? 



65. When does this notification occur? 



66. Who is responsible for notifying families selected for the sample? (At what level?) 



67. What information/documentation are selected families asked to provide? 



a. What about students who are categorically eligible for meals? 



68. Is there a deadline for families to submit the information requested? 
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69. Who is responsible for receiving verification documentation (and at what level)? Is the 
information transferred to other offices to be reviewed? 



Reviewing Documentation: 

70. What process is used to review income verification documentation? 

71. What information is used to confirm, adjust, or terminate eligibility? How is the 
information used to do this? 



72. What staff are involved in this process? What level do they work at — district or school? 

73. When does this work take place? 

74. What happens if a family submits incomplete documentation during the verification 
process? 

75. How do you handle families who do not respond to the request for documentation? 

76. How are families notified about the results of the verification? 



77. Who performs this task (and at what level)? 

78. When does notification occur? 



79. How are the results of the verification process communicated from the district level to 
individual schools? How is this information made available to staff in the cafeteria? 



80. What data and records are kept at the district and school levels from the verification 
process? 
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Additional Verification (GV Pilot Only): 

81. Who checks the error rate of the initial verification? 



82. How and when is this done? 



83. If the error rate is above 25 percent, how does the district select half of the remaining 
households for the second round of verification? 



84. Did you complete a second round of verifications in 2001-2002? If so, how many 
applications were verified during the second round? 

85. Does the second round of verification differ from the first round in terms of requesting, 
receiving, or reviewing documentation? 

86. Do the same staff perform these tasks as performed the original round of verifications? 

87. Does the second round of verifications take the same, more, or less time than the first? 



88. When (during which months) does the second round occur? 

89. Did you have a third round of verifications during 2001-2002? If so, how many 
applications were verified? 

90. If a third round of verifications is required, how does the process differ from the second 
round of verifications? 



91. When during the school year does the third round of verifications occur, if required? 



General: 

92. We’ve discussed the steps involved in sampling, contacting families, receiving and 
reviewing documentation, and notifying families of their eligibility status during the 
verification process. Are there additional tasks that we have not talked about, that are 
involved in the verification process? (If so, list activities, staff, level, and time frame) 
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II. COUNTS 



for October 2002: 

Number of applications received (may be per child or per household, depending on 
district) — if possible, divide into applications received during initial period, and then 
monthly thereafter 

Number of written applications approved F/RP — same as above 

Number of students/applications verified in 2001-2002 school year, by verification round if 
possible 



III. CHANGES IN PROCEDURES SINCE 1999 

The next series of questions concern changes in the application and verification procedures since 
the fall of 1999 (the year before the pilot program began, for pilot districts). 

93. Did the district use direct certification in 1999? 

94. Has the process of direct certification changed since 1999? If so, how? 

95. How was outreach and dissemination of NSLP applications different in 1999 than it is 
now? 

96. How have the materials that are sent to households regarding the NSLP changed since 
1999? 

97. Is the process of receiving applications different? If so, how? 

98. Are applications received and reviewed by the same types of staff? 

99. In what ways has the process of reviewing applications changed? 

100. (UFD Pilot Only): How has the process of collecting income documentation for all 
applications affected reviewing applications? 

101. How has the verification process changed since 1999? 
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102. (UFD Pilot Only): What level of resources did verification require from your district in 
1999? How does this compare with the resources required to collect income 
documentation for all applications under the pilot? (level of resources = time, money, 
effort) 

103. (UFD) How was the initial verification sample selected? When was this done and by 
whom? 

104. (UFD) How did you notify families that they’d been selected for verification? Who did 
this? 

105. (UFD) What info/documentation were selected families asked to provide? 

106. (UFD) Was there a deadline given? Approximately how long did they have? 

107. (UFD) How did you go about reviewing the verification documentation, and who did 
this? 

108. (UFD) What sorts of documentation would families send to confirm eligibility? 

109. (UFD) What happened if a family submitted incomplete documentation? 

110. (UFD) How did you handle families who did not respond to the request for 
documentation? 

111. (UFD) How were families notified about the results of verification (and who did this 
task)? 

112. (UFD) How were the results of verification communicated from the district level to the 
individual schools? How is the information made available to staff in the cafeteria? 

113. (GV Pilots Only): How many rounds of verification have you performed in the years 
since you’ve started the pilot? How has the level of resources required to perform 
verification changed since 1999? 



III. PERCEPTIONS 

114. (Pilots only) What benefits have been realized by using the pilot program in your 
district? 



115. (Pilots only) Were these benefits expected or unexpected? 



11 6. (Pilots only) Are there any benefits you’d hoped the pilot would provide but have not 
been realized? 
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1 17. (Pilots only) What have been the main challenges involved in using the pilot? 

a. How frequently do these challenges occur, and how severe are they? 

11 8. (Pilots only) How have you handled these challenges? 

119. (Pilots only) Has the pilot affected the volume of families applying for free or reduced 
price NSLP meals? If so, why/how do you think the pilot has had this effect? 

120. (Pilots only) Has the pilot affected the number of families being approved to receive 
free or reduced price meals through the NSLP? If so, why/how do you think the pilot 
has had this effect? 

121. (Pilots only) Has the pilot affected the likelihood that an ineligible student will apply for 
and be approved for F/RP meals? How did it do so? 

122. (Pilots only) Has the pilot affected the likelihood that an eligible student would be 
approved for F/RP meals? 

123. (ALL DISTRICTS) How well do households understand eligibility requirements for the 
NSLP? 



124. (ALL DISTRICTS) What types of problems do families encounter in the application 
and verification processes that they bring to the attention of SFA staff? How are these 
problems handled? 
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CUSTOMIZED COST WORKSHEETS 




NSLP APPLICATIONA^ERIFICATION PILOT EVALUATION 



TYPICAL UP-FRONT DOCUMENTATION DISTRICT 

TIME SPENT CONDUCTING DIRECT CERTIFICATION 
[WORK OCCURRING IN AUGUST] 



Activity 



Person 1 Person 2 Other Staff Total Comments on 

Hours Hours Hours Hrs Changes since 

(list position/name) 2000-2001 



Determining if 
children on state 
list are enrolled 
in district, and 
preparing/sending 
notification letters 



Updating files, 
entering information 
to computer 



Other (please describe): 
Phone calls 



Total Hours: 

Total Labor Cost: 

Estimated budget for all non-labor costs (printing, mailing, data processing, etc.) associated with 
conducting direct certification: 
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NSLP APPLICATIONA^ERIFICATION PILOT EVALUATION 



TYPICAL UP-FRONT DOCUMENTATION DISTRICT 

TIME SPENT PROCESSING APPLICATIONS 
[WORK OCCURRING FROM AUGUST THROUGH OCTOBER] 



Activity 



Person 1 Person 2 Other Staff Total Comments on 

Hours Hours Hours Hrs Changes since 

(list position/name) 2000-2001 



Preparing 
Applications 
for distribution 
hy schools 

Checking returned 
Applications 
for completeness 

Generating letters 
to families with 
incomplete 
applications 

Phone calls to 
families 

with incomplete 
applications 

Determining 
eligibility 
of complete 
applications 
using income 
documentation 

Entering 
approved 
students 
to list 

Filing hard 
copy apps 

Generating/ 

Sending 

letters 
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Activity 



Person 1 Person 2 Other Staff Total Comments on 

Hours Hours Hours Hrs Changes since 

(list position/name) 2000-2001 



Preparing/sending 
eligibility 
information/ 
lists to schools 

Answering 
questions/phone 
calls about why 
families were 
denied 

Dealing with any 
questions or 
problems at 
the schools 

Other (please describe): 



Total Hours: 

Total Labor Cost: 

Estimated budget for all non-labor costs (copying, printing, mailing, data processing, etc.) 
associated with application activities occurring in the beginning of the school year, from August 
through October: 

Estimated number of applications received each month after main period of application activity 
(or, after October): 
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NSLP APPLICATIONA^ERIFICATION PILOT EVALUATION 



TYPICAL GRADUATED VERIFICATION DISTRICT 



TIME SPENT CONDUCTING DIRECT CERTIFICATION 
[MOST WORK OCCURS IN AUGUST] 



Activity 


Person 1 


Person 2 


Total 


Comments on 




Hours 


Hours 


Hours 


Changes Since 
1999-2000 



Determining if children 
on state list are enrolled 
in district 



Updating files 



Preparing notification 
letters to be sent 
(stuffing envelopes, 
writing mail labels) 



Other (please describe): 
Subsequent checking of list 



Totals Hours: 

Total Costs: 

Hourly Salary = for person 1 and for person 2. 

Estimated budget for all non-labor costs (printing, mailing, data processing, etc.) associated with 
conducting direct certification: 
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NSLP APPLICATIONA^ERIFICATION PILOT EVALUATION 



TYPICAL GRADUATED VERIFICATION DISTRICT 



TIME SPENT PROCESSING APPLICATIONS 
[WORK OCCURRING FROM AUGUST THROUGH OCTOBER] 



Activity 


Person 1 


Person 2 


Total 




Hours 


Hours 


Hrs. 



Processing apps 
(entering info to computer, 
determining eligibility, 
filing hard copy apps) 



Mailings to families 
With incomplete apps 



Phone Calls 



Notification Letters 
(generating letters, stuffing 
envelopes, addressing labels) 



Transferring eligibility 
Information to schools 



Totals 

Costs: 

Hourly Salary = for person 1 and for person 2. 

Estimated budget for all non-labor costs (copying, printing, mailing, data processing, etc.) 
associated with application activities occurring in the beginning of the school year, from August 
through October: 

Estimated number of applications received each month after main period of application activity 
(or, after October): 
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NSLP APPLICATIONA^ERIFICATION PILOT EVALUATION 



TYPICAL GRADUATED VERIFICATION DISTRICT 



TIME SPENT VERIFYING APPLICATIONS, BY ROUND 



Activity 


Person 1 


Person 2 


Total 


Comments on 




Hours 


Hours 


Hrs. 


Changes Since 
1999-2000 



ROUND 1 (3%): 

Selecting sample/ 
sending letters 

Mailings to families not 
replying or submitting 
incomplete documentation 

Phone calls 

Processing verification 
documentation (entering 
it to the computer, 
calculating gross income, etc.) 

Notifying families of verification results 



ROUND 2: 

Selecting sample/ 
sending letters 

Mailings to families not 
replying or submitting 
incomplete documentation 

Phone calls 

Processing verification 
documentation (entering 
it to the computer, 
calculating gross income, etc.) 

Notifying families of verification results 
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Activity 


Person 1 


Person 2 


Total 


Comments on 




Hours 


Hours 


Hrs. 


Changes Since 
1999-2000 



ROUND 3: 

Selecting sample/ 
sending letters 

Mailings to families not 
replying or submitting 
incomplete documentation 

Phone calls 

Processing verification 
documentation (entering 
it to the computer, 
calculating gross income, etc.) 

Notifying families of verification results 



Total Hours: 

Total Costs: 

Hourly Salary = for person 1 and for person 2. 

Estimated budget for all non-labor costs (copying, printing, mailing, data processing, phone 
calls, etc.) associated with verification activities, by round: 

Round 1: 

Round 2: 

Round 3: 
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NSLP APPLICATIONA^ERIFICATION PILOT EVALUATION 



TYPICAL COMPARISON DISTRICT 

TIME SPENT PROCESSING APPLICATIONS 
[FOR WORK OCCURRING DURING THE MAIN PERIOD OF APPLICATION 
ACTIVITY, FROM AUGUST THROUGH SEPTEMBER] 



Activity 


Number of Hours 




Comments on 




Person 1 


School 

Staff 


Changes Since 
1999-2000 



Distributing applications to students 

Processing applications 
(reviewing for completeness, 
entering data into computer, 
determining eligibility, 
filing hard copy applications) 

Mailings to families submitting 
incomplete applications 

Notifying applicants of eligibility 
determination (generating letters, 
stuffing envelopes, addressing labels, etc.) 

Transferring eligibility information 
to school cafeteria staff 

Other (please describe): 



Hourly pay rate 
Hourly fringe benefits 



Estimated budget for all non-labor costs (copying, printing, mailing, data processing, etc.) 
associated with application activities occurring in the beginning of the school year, during the 
main period of application activity: 

Estimated number of applications received during the main period of application activity: 

Estimated number of applications received after the main period of application activity: 
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NSLP APPLICATIONA^ERIFICATION PILOT EVALUATION 



TYPICAL COMPARISON DISTRICT 
TIME SPENT VERIFYING APPLICATIONS 



Activity 



Number of Hours 
Person 1 School 

Staff 



Comments on 
Changes Since 



1999-2000 



Selecting sample 

Sending initial notification letters 

Phone calls or mailings to fa mi lies 
not replying or submitting 
incomplete documentation 

Processing verification 
documentation 
(entering into computer, 
calculating gross income, etc.) 

Notifying families of verification results 

Notifying schools of changes in status 

Other (please describe): 



Estimated budget for all non-labor costs (copying, printing, mailing, data processing, phone 
calls, etc.) associated with verification activities: 
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